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Parallel Dynamics of Continuous Hopfield Model Revisited 
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We have applied the generating functional analysis (GFA) to a continuous Hopfield model. 
We have also confirmed that the GFA predictions in some typical cases exhibit good consistency 
with computer simulation results. When a retarded self-interaction term is omitted, the GFA 
result becomes identical to that obtained using the statistical neurodynamics as well as the 
case of the sequential binary Hopfield model. 
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In the field of information theory, various impor- 
tant inference problems are described by continuous 
nonlinear systems which include disorder. Some exam- 
ples are code-division multiple-access (CDMA), 1-4 error- 
correcting codes, 5-10 lossy compression, 11-16 and com- 
puter vision. 17 These techniques play an important role 
in our modern life. 

Equilibrium statistical mechanics is used to evaluate 
the performance of information processing problems that 
are formulated using statistical models with a large de- 
gree of freedom. These problems can be reformulated 
as a computation of marginal probabilities on factor 
graphs. Some iterative methods, which can be used to 
calculate marginal probabilities approximately, e.g., be- 
lief propagation, give us effective algorithms as iterative 
equations with parallel update. In general, the theoret- 
ical treatment of the dynamics of disordered systems is 
complicated compared with that of equilibrium statis- 
tical mechanics. It's important to understand the dy- 
namical property of iterative algorithms such as a belief- 
propagation-based one in order to improve its perfor- 
\ mance. 

In this study, we treat the continuous Hopfield model 18 
with the zero-temperature parallel dynamics as a typi- 
cal example. There are many studies using the generat- 
ing functional analysis (GFA) 19 of the Hopfield model 
with various dynamics. The GFA allows us to study 
the dynamics of an infinitely large disordered system 
in an exact way. For instance, Crisanti and Sompolin- 
sky analyzed the asymmetric spherical-state Hopfield 
model with Langevin dynamics. 20 Crisanti and Som- 
polinsky also analyzed an asymmetric Ising system with 
sequential-updated Glauber dynamics. 21 Rieger analyzed 
the binary-state Hopfield model with sequential-updated 
Glauber dynamics. 22 Gardner et al. analyzed the Hop- 
field model with the zero temperature parallel dynam- 
ics. 23 Their approach can be used to calculate magneti- 
zation after several time steps but not to treat continuous 
states. 

Kawamura et al. have found the sequential binary Hop- 
field model shows chaotic behaviors. 24 ' 25 In the case of 
the sequential binary Hopfield model with the Glauber 
dynamics, Kawamura and Okada have shown that the 
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GFA result and the statistical neurodynamics (SND) 26 
result are identical. 27 Bollc et al. have investigated the 
relationship between the GFA and the SND using the 
binary Hopfield model with the Glauber dynamics. 28 
Hashiguchi and Kawamura have studied the sequential 
continuous Hopfield model by using the SND. 29 However, 
the relationship between the GFA and the SND in the 
continuous Hopfield model with parallel dynamics has 
remained unclear so far. Therefore, we here study this 
relationship. We apply the Langevin dynamics 20 at zero 
temperature as the parallel dynamics to treat the contin- 
uous state. This dynamics has been applied to analysis 
of the batch minority games, 30 etc. To confirm the va- 
lidity of our analysis, we have performed some computer 
simulations for some typical conditions. 

We consider a nonlinear system that consists of A*" con- 
tinuous variables <Ji(t) £ 1 at time t. In this paper, we 
treat the following updating rule: 

o-i(i + 1) - f(ui(t)), Ui {t) = hi{a{t)) + 6i{t) (1) 

where / : R — » M is an arbitrary function that 
called a transfer function. Here, Ui(t) is called a lo- 
cal field, that includes the function hi(<r(t)) of <r(t) — 
'(<7i(i),- • • ,<T]\r(i)) and a time-dependent external field 
9i (t) to calculate a response function. The operator > de- 
notes the transpose. We analyze the transient dynamics 
in the thermodynamical limit, i.e., N — > oo. The Glauber 
dynamics can apply only the Ising system. Therefore, we 
first define the following deterministic updating rule for 
the variable o~i(t) at time t as 

p[tTi(t + l)lo-(i)] = S[o-i(t + 1) - f( Ui (t))], (2) 

which can be regarded as the Langevin dynamics at 
zero temperature, where S denotes the Dirac delta fac- 
tion. This dynamics represents Markovian dynamics, so 
the path probability p[cr(0), ■ • • , <x(i)] is simply given by 
the products of the individual transition probabilities 
p[er(s + l)|er(s)] = Yli=i P^iis + tyl&i 3 )] °f tnc chain: 

t— l 

p[a(0), ■■■ , v(t)] = p[<r(0)] J] p[<r(s + l)\a(s)}. (3) 

s=Q 

If we assume that we know the initial microscopic 
state precisely, it allows for a factorized distribution as 

p[*(o)] = n£iftfa(o)]- 
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To analyze the transient dynamics of the continuous 
nonlinear system, we define a generating functional Z[i/>] 
as follows: 

zm = [ (f[d*( S ))p[*(o),--- ,*(t)} 

J R (t+l)iV \fJ- Q J 



N 



x exp I — i ^ er(s) • ip(s) 

^ s=0 



(4) 



where <x(s) = t(ai(s), • • • , ctjv(s)) and t/>(s) = 
t (V'i(s), • • • ,^>jv(s))- In a familiar way, 19 one can ob- 
tain from Z[ij)] all averages of interest by differentiation, 
i.e., spin-averages (or overlap), correlation- functions, and 
response- functions: 



rrii(s) = {<Ji(s)} = i lim 



dZ[ip] 



dZ[i/>] 



Cij (s,s>) = =-$* ma)ma ,y 



Gij(s,s') = 



= i lim 



dZ[ij>] 



(5) 



d6j(s') v^o 

where ( ) and ~ denote the average with respect to the 
path probability p[<x(0), • • • ,(r(t)] and the average over 
the disorder that is included in the generating functional 
Z[ip], respectively. 

In the continuous Hopfield model, the local field is de- 
fined as follows: 

N aN 

Ui ( S ) = -w*) + 0i( S ), in = -Yl ( 6 ) 

j=l li=l 

where a is the loading rate. The pattern components 
{£f} follow an identical and independent distribution 



p(0 



^(e-i) + ^r + i)- 



(7) 



Performing an appropriate gauge transformation, we 
can put = 1 (Vi) without a loss of generality. 
We first separate the local field u,(s) by inserting 
the following 5-function: 1 = J nl=o IIi=i dui(s)dui(s) 
exp[iUi(s){ui(s)-Ylf=i Jij<Tj(s)-0i(s)}]- Averaging the 
generating functional over the disorder {£f }, we separate 
the relevant one-time and two-time order parameters: 

m (s) = ^E J =i cr i( s )> k (s) = ^E^Ii"i( s )> ?( s > s ') = 
wY, I iLi (T *( s ) a i( s '), Q(s,s') = ^Eili^( s )^( s ')> and 

K(s,s') = EiLi °"i( s )wi(s'). We then obtain the 
disorder-averaged generating functional 



m = j 



dmdrhdkdkdqdqdQdQdKdKe 



JV(*+$+J2) 



(8) 



where 

t-i 

* = i^2{m(s)m(s) + k(s)k(s) - m(s)k(s)} 

s=0 



t-i t-i 

+ z E Z>(*> s 'W s > s ') + & s > s 'M s < s ') 

s=0 s'=0 

+ k(s,s')K( S ,s')}, (9) 



l^E ln 



] J ( /t (.-:• )p[a(0)] / <5/i<5/i 
, H- 1) - f(h(s))] 



s=0 



s=0 s'=0 



x exp 

+ Q(s, s')^(s)/i(s') + K(s, s')a(s)h(s')} 
t-i 

+ iJ2ks){h( s )-e l ( s )-k( s )£} 



s=0 

t-1 



1 1 

-i5^CT(a)m(s)^-*5^<T(s)0j(s) \ 

s=0 s=0 -I 



(10) 



and 



n = -m 



exp 



N t-1 



M= 2 s=0 



Bp (*)»/*(*)} 

JEEE^^mm*') 



aAT t-1 t-1 



M= 2 s=0 s'=0 

+ x„(8)K(8, s')v^s') + y^(s)K(s, s')^(s') 



+ iv( s )<?( s ' s ')i//x(s') 



(11) 



with = ni= ^?, ^ = ni= ^, = 
nl= n;=2 ^ n = uTJou;^ ^0- 

The disorder-averaged generating functional is for 
N — > oo dominated by a saddle-point. Applying the 
saddle-point method, we arrive at the following result. 
Some macroscopic variables are found to vanish at the 
relevant physical saddle-point. The remaining variables 
can all be expressed in terms of three macroscopic ob- 
servables, namely, the overlaps m(s), the single-site cor- 
relation functions C(s,s') and the single-site response 
functions G(s, s'). We then arrive at the following saddle- 
point equations: 

m(s) = «*(*)», 

C(s,s>) = H*Ms')», (12) 

a~ 1 / 2 ([a{8)(Rr 1 v)(8')}, for 8 > s ' 
0, for s < s'. 

The average over the effective path measure is given by 

t 



G(s,s') 



{ 9 {a,v))) = f Vv f (f[da(s))g(a,v) 

t-1 

xp[a(0)]l[6[a(s + l)-f(u(s))}, (13) 



s=0 
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Fig. 1. The first few time steps of the dynamics of the binary 
Hopfield model predicted using the GFA with a = 0.12 (cir- 
cles and solid lines) and a = 0.20 (circles and dashed lines). 
The transfer function is f(x) = sgn(x). Computer simulations 
(crosses) were performed with N = 10, 000 from 10 experiments. 




Fig. 2. The first few time steps of the dynamics of the continuous 
Hopfield model predicted using the GFA with a = 0.12 (circles 
and solid lines) and a = 0.20 (circles and dashed lines). The 
transfer function is f(x) = — tanh[7(:r + k)] tanh[7:r] tanh[7(x — 
k)] with 7 = 5 and k = 1. Computer simulations (crosses) were 
performed with N = 10, 000 from 10 experiments. 



with 



u(s) = m(s) + 6(s) + y/av(s) + (T<r)(s), 
R=(l-G)- 1 C(l-^G)-\ 
T = a(l-G)- 1 G, 

R- 



(14) 



and Vv = \2irR\- 1 l 2 dve-i v - H * v . Here, C, G, R 
and r are matrices having matrix elements C(s,s'), 
G(s,s'), R(s,s'), and T(s,s'), respectively, <x is the vec- 
tor V(0), • • • , o-(i)). The terms (R- 1 v)(s) and (IV) (s) 
denote the sth element of the vectors R _1 v and IV, 
respectively. The matrix 1 denotes a unit matrix. Equa- 
tions (12)-(14) completely describe the dynamics of this 
continuous nonlinear system. In the limit where t — > oo, 
the term (IV) (s) in cq. (14) can be regarded as the On- 
sager reaction term in equilibrium statistical mechanics. 

In most applications in the field of information theory, 
the initial state is usually given by a discrete distribution 



having the form 



K 



(15) 



k=i 



where > (Vfc), J2k=i Ck = 1 an( ^ e R (Vfc). 
When the initial probability is given by the form of eq. 
(15), the spin integral included in eq. (13) becomes a 
simple summation and the average over the effective path 
measure can be simplified to 



((9(<r,v))) 



K 



Cfe 



a(0)=A*,<7(l)=/(u(0)),-,a(t)=/(«(t-l)) 

(16) 

The spin integral in cq. (13) can be performed in this 
system, therefore the effective single-spin dynamics a(s + 
1) = f(u(s)) appears in eq. (16). The initial state is cho- 
sen stochastically from the initial distribution of eq. (15), 
but dynamics is deterministic. The deterministic path of 
the initial state <r(0) = Afe appears with a probability 
Cfe. The average over the effective path measure can be 
calculated from K deterministic paths. 

To validate the GFA, we performed computer simula- 
tions. We considered a case with the initial distribution 
p[a{0)} = i(l + m(0))<5(cT(0)-l) + i(l-m(0))(5(cr(0) + l) 
and 9(s) = 0. The first few time steps of the dynamics of 
the binary Hopfield model, which has the transfer func- 
tion f(x) = sgn(x), predicted using the GFA with a = 
0.12 and a = 0.20 are shown in Fig. 1. The computer sim- 
ulations were performed with N — 10, 000 from 10 exper- 
iments. Excellent agreement was found between the pre- 
dictions of the GFA and the computer simulation results. 
The first few time steps of the dynamics of the continuous 
Hopfield model predicted using the GFA with a = 0.12 
and a = 0.20 are shown in Fig. 2. The transfer func- 
tion is f(x) = — tanh[7(x + k)] tann^x] tanh[7(x — k)] 
with 7 = 5 and k = 1. Note that the parameter 7 is 
not the inverse temperature but an arbitrary parameter. 
In this case, computer simulations were also performed 
with TV = 10, 000 from 10 experiments. When the trans- 
fer function / is continuous, the GFA exhibited excellent 
consistency with the computer simulation results. 

The relationship between the GFA and the SND is 
as follows. The SND analysis ignores the Onsager reac- 
tion term. 26 In the binary-state parallel-update sequen- 
tial processing Hopfield model, which does not have the 
Onsager reaction term, Kawamura and Okada showed 
that the GFA result was identical to the SND result. 27 
According to their discussion, if we neglect the Onsager 
reaction term, we can show that GFA result is equiva- 
lent to the SND result. When we neglect the Onsager 
reaction term in the GFA result of the continuous Hop- 
field model, which has the Onsager reaction term, the 
response function becomes 

G ^ S '^7^ I V < s - 1 ) I d <s) a(s)8[a(s) 
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- f{m{s - 1) + 9{a - 1) + y/ov(s - 1))], 



where X>v(s - 1) = (2tt.R(s - l,s - 1)) 



(17) 

-1/2 X 



-it>(s-l) 2 /K(s-l,s-l) 



This means that the response 



function becomes zero except for G(s,s — 1). Therefore, 
the (s, s')th clement of the matrix G™ has the simple 
form G n (s, s') = 5 s ^ +n n"=o G ( s ~ r,s - t - 1). When 
the Onsager reaction term is neglected, eq. (14) can be 
represented in the recurrence relation form 

R(s,s') =C(s,s') 

+ G(s, s - l)G(s', s' - l)R(s - 1, a' - 1) 



+ ^C(A,s') J2 G ( 

A+l 



A=0 



1) 



+ ]TC( S ,A) Yl G(r,r-1), 

T = A+1 



(18) 



A=0 



by using the identity (1 
order parameters become 



G)- 1 = 1 + £„=i G". The 

m(s) = J Dzf(u(s-1)), 
C(s, s') = J DuDvDz f(u(s - 1, s' 



G(s,s-1) 



x /(«(«' 
1 



l,s 



1)) 
1)), 



(19) 



^JaR{s- l,s- 1) 



where u(s) = m(s) +z-y/a:i?(s, s) and u(s,s') = m(s) 
+u-^/a[R(s, s) — R(s, s')] +zy / aR(s, s'). This result ob- 
tained using eqs. (18)-(19) coincides with the SND re- 
sult. 26 Namely, the SND result can be derived from the 
GFA result by neglecting the Onsager reaction term. 

In summary, we applied the generating functional anal- 
ysis to the continuous Hopficld model. The predictions of 
the GFA are in excellent agreement with computer sim- 
ulation results for any conditions and transfer function. 
When the retarded self-interaction term is omitted, the 
GFA result becomes identical to the SND result. The 
GFA can also be applied to any continuous nonlinear 
system having the form of cq. (2) which includes some 



belief-propagation-based algorithms, e.g., multiuser de- 
tection problems in mobile communication systems. 
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